1. Prior Art on Out of Position Occupants and Rear Facing Child Seats
Whereas thousands of lives have been saved by airbags, a large number of people have also been injured, some seriously, by the deploying airbag, and thus significant improvements need to be made in this regard. As discussed in detail in copending patent applications 08/040,978 and 08/239,978 cross-referenced above, for a variety of reasons vehicle occupants may be too close to the airbag before it deploys and can be seriously injured or killed as a result of the deployment thereof. Also, a child in a rear facing child seat that is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as first publicly disclosed in Breed, D. S. xe2x80x9cHow Airbags Workxe2x80x9d presented at the International Conference on Seatbelts and Airbags in 1993, in Canada, occupant position sensing and rear facing child seat detection is required.
Initially these systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded airbag deployments when a front seat is unoccupied. However, airbags are now under development to protect rear seat occupants in vehicle crashes and all occupants in side impacts. A system will therefore be needed to detect the presence of occupants, determine if they are out-of-position and to identify the presence of a rear facing child seat in the rear seat. Future automobiles are expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance and possible harm of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident needlessly.
Inflators now exist which will adjust the amount of gas flowing to the airbag to account for the size and position of the occupant and for the severity of the accident. The vehicle identification and monitoring system (VIMS) discussed in patent application Ser. No. 08/239,978 will control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. The instant invention is an improvement on that VIMS system and uses an advanced optical system comprising one or more CCD (charge coupled device) arrays and a source of illumination combined with a trained neural network pattern recognition system.
The need for an occupant out-of-position sensor has been observed by others and several methods have been disclosed in U.S. patents for determining the position of an occupant of a motor vehicle. Each of these systems, however, has significant limitations. In White et al. (U.S. Pat. No. 5,071,160), for example, a single acoustic sensor and detector is disclosed and, as illustrated, is mounted lower than the steering wheel. White et al. correctly perceive that such a sensor could be defeated, and the airbag falsely deployed, by an occupant adjusting the control knobs on the radio and thus they suggest the use of a plurality of such sensors.
Mattes et al. (U.S. Pat. No. 5,118,134) disclose a variety of methods of measuring the change in position of an occupant including ultrasonic, active or passive infrared and microwave radar sensors, and an electric eye. Their use of these sensors is to measure the change in position of an occupant during a crash and use that information to access the severity of the crash and thereby decide whether or not to deploy the airbag. They are thus using the occupant motion as a crash sensor. No mention is made of determining the out-of-position status of the occupant or of any of the other features of occupant monitoring as disclosed in the above cross-referenced patent applications. It is interesting to note that nowhere does Mattes et al. discuss how to use active or passive infrared to determine the position of the occupant. As pointed out in the above cross-referenced patent applications, direct occupant position measurement based on passive infrared is probably not possible and, until very recently, was very difficult and expensive with active infrared requiring the modulation of an expensive GaAs infrared laser. Since there is no mention of these problems, the method of use contemplated by Mattes et al. must be similar to the electric eye concept where position is measured indirectly as the occupant passes by a plurality of longitudinally spaced-apart sensors.
The object of an occupant out-of-position sensor is to determine the location of the head and/or chest of the vehicle occupant relative to the airbag since it is the impact of either the head or chest with the deploying airbag which can result in serious injuries. Both White et al. and Mattes et al. disclose only lower mounting locations of their sensors that are mounted in front of the occupant such as on the dashboard or below the steering wheel. Both such mounting locations are particularly prone to detection errors due to positioning of the occupant""s hands, arms and legs. This would require at least three, and preferably more, such sensors and detectors and an appropriate logic circuitry which ignores readings from some sensors if such readings are inconsistent with others, for the case, for example, where the driver""s arms are the closest objects to two of the sensors.
White et al. also disclose the use of error correction circuitry, without defining or illustrating the circuitry, to differentiate between the velocity of one of the occupant""s hands as in the case where he/she is adjusting the knob on the radio and the remainder of the occupant. Three ultrasonic sensors of the type disclosed by White et al. might, in some cases, accomplish this differentiation if two of them indicated that the occupant was not moving while the third was indicating that he or she was. Such a combination, however, would not differentiate between an occupant with both hands and arms in the path of the ultrasonic transmitter at such a location that they were blocking a substantial view of the occupant""s head or chest. Since the sizes and driving positions of occupants are extremely varied, trained pattern recognition systems, such as neural networks, are required when a clear view of the occupant, unimpeded by his/her extremities, cannot be guaranteed.
Fujita et al., in U.S. Pat. No. 5,074,583, illustrates another method of determining the position of the occupant but do not use this information to suppress deployment if the occupant is out-of-position. In fact, the closer that the occupant gets to the airbag the faster the inflation rate of the airbag is according to the Fujita patent, which thereby increases the possibility of injuring the occupant. Fujita et al. do not measure the occupant directly but instead determine his or her position indirectly from measurements of the seat position and the vertical size of the occupant relative to the seat. This occupant height is determined using an ultrasonic displacement sensor mounted directly above the occupant""s head.
As discussed above, the optical systems described herein are also applicable for many other sensing applications both inside and outside of the vehicle compartment such as for sensing crashes before they occur as described in copending patent application Ser. No. 08/239,978 cross-referenced above, for a smart headlight adjustment system and for a blind spot monitor.
2. Definitions
The use of pattern recognition is central to the instant invention as well as those cross-referenced patent applications above. Nowhere in the prior art is pattern recognition that is based on training, as exemplified through the use of neural networks, mentioned for use in monitoring the interior or exterior environments of the vehicle. xe2x80x9cPattern recognitionxe2x80x9d as used herein will mean any system which processes a signal that is generated by an object, or is modified by interacting with an object, in order to determine which one of a set of classes that the object belongs to. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally electrical signals coming from transducers which are sensitive to either acoustic or electromagnetic radiation and, if electromagnetic, they can be either visible light, infrared, ultraviolet or radar. A trainable or a trained pattern recognition system as used herein means a pattern recognition system which is taught various patterns by subjecting the system to a variety of examples. The most successful such system is the neural network.
To xe2x80x9cidentifyxe2x80x9d as used herein will mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all rear facing child seats, one containing all human occupants, or all human occupants not sitting in a rear facing child seat depending on the purpose of the system. In the case where a particular person is to be recognized, the set or class will contain only a single element, i.e., the person to be recognized.
An xe2x80x9coccupying itemxe2x80x9d of a seat may be a living occupant such as a human being or a dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
In the description herein on anticipatory sensing, the term xe2x80x9capproachingxe2x80x9d when used in connection with the mention of an object or vehicle approaching another will mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact with a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The xe2x80x9ctargetxe2x80x9d vehicle is the vehicle that is being impacted. This convention permits a general description to cover all of the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
3. Pattern Recognition Prior Art
Japanese patent 3-42337 (A) to Ueno discloses a device for detecting the driving condition of a vehicle driver comprising a light emitter for irradiating the face of the driver and a means for picking up the image of the driver and storing it for later analysis. Means are provided for locating the eyes of the driver and then the irises of the eyes and then determining if the driver is looking to the side or sleeping. Ueno determines the state of the eyes of the occupant rather than determining the location of the eyes relative to the other parts of the vehicle passenger compartment. Such a system can be defeated if the driver is wearing glasses, particularly sunglasses, or another optical device which obstructs a clear view of his/her eyes. Pattern recognition technologies such as neural networks are not used.
U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules to isolate the eyes and mouth of a driver and uses this information to permit the driver to control the radio, for example, or other systems within the vehicle by moving his eyes and/or mouth. Ando uses natural light and illuminates only the head of the driver. He also makes no use of trainable pattern recognition systems such as neural networks, nor is there any attempt to identify the contents of the vehicle nor of their location relative to the vehicle passenger compartment. Rather, Ando is limited to control of vehicle devices by responding to motion of the driver""s mouth and eyes.
U.S. Pat. No. 5,298,732 to Chen also concentrates in locating the eyes of the driver so as to position a light filter between a light source such as the sun or the lights of an oncoming vehicle, and the driver""s eyes. Chen does not explain in detail how the eyes are located but does supply a calibration system whereby the driver can adjust the filter so that it is at the proper position relative to his or her eyes. Chen references the use of automatic equipment for determining the location of the eyes but does not describe how this equipment works. In any event, there is no mention of monitoring the position of the occupant, other that the eyes, of determining the position of the eyes relative to the passenger compartment, or of identifying any other object in the vehicle other than the driver""s eyes. Also, there is no mention of the use of a trainable pattern recognition system.
U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle. Faris locates the eyes of the occupant by the use of two spaced apart infrared cameras using passive infrared radiation from the eyes of the driver. Again, Faris is only interested in locating the driver""s eyes relative to the sun or oncoming headlights and does not identify or monitor the occupant or locate the occupant relative to the passenger compartment or the airbag. Also, Faris does not use trainable pattern recognition techniques such as neural networks. Faris, in fact, does not even say how the eyes of the occupant are located but refers the reader to a book entitled Robot Vision (1991) by Berthold Horn, published by MIT Press, Cambridge, Mass. Also, Faris uses the passive infrared radiation rather than illuminating the occupant with active infrared radiation or in general electromagnetic radiation as in the instant invention.
The use of neural networks as the pattern recognition technology is central to this invention since it makes the monitoring system robust, reliable and practical. The resulting algorithm created by the neural network program is usually only a few lines of code written in the C computer language as opposed to typically hundreds of lines when the techniques of the above patents to Ando, Chen and Faris are implemented. As a result, the resulting systems are easy to implement at a low cost making them practical for automotive applications. The cost of the CCD arrays, for example, have been prohibitively expensive until very recently rendering their use for VIMS impractical. Similarly, the implementation of the techniques of the above referenced patents requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than $5.
The present invention uses sophisticated trainable pattern recognition capabilities such as neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction. An example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. xe2x80x9cAnalysis of Hidden Units in a Layered Network Trained to Classify Sonar Targetsxe2x80x9d, Neural Networks, Vol. 1. pp 75-89, 1988, and xe2x80x9cLearned Classification of Sonar Targets Using a Massively Parallel Networkxe2x80x9d, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled xe2x80x9cProcess and Apparatus for the Automatic Detection and Extraction of Features in Images and Displaysxe2x80x9d to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled xe2x80x9cArtificial Neuron and Method of Using Same and U.S. patent application Ser. No. 08/076,601 entitled xe2x80x9cNeural Network and Method of Using Samexe2x80x9d to Wang, S. T. Other examples include U.S. Pat. No. 5,235,339 (Morrison et al.), U.S. Pat. NO. 5,214,744 (Schweizer et al), U.S. Pat. No. 5,181,254 (Schweizer et al), and U.S. Pat. No. 4,881,270 (Knecht et al). All of the above references are included herein by reference.
4. Optics
Optics can be used in several configurations for monitoring the interior of a passenger compartment of an automobile. In one known method, a laser optical system uses a GaAs infrared laser beam to momentarily illuminate an object, occupant or child seat, in the manner as described and illustrated in FIG. 8 of the copending patent application Ser. No. 08/040,978 cross-referenced above. The receiver can be a charge coupled device or CCD, (a type of TV camera) to receive the reflected light. The laser can either be used in a scanning mode, or, through the use of a lens, a cone of light can be created which covers a large portion of the object. In these configurations, the light can be accurately controlled to only illuminate particular positions of interest within the vehicle. In the scanning mode, the receiver need only comprise a single or a few active elements while in the case of the cone of light, an array of active elements is needed. The laser system has one additional significant advantage in that the distance to the illuminated object can be determined as disclosed in the Ser. No. 08/040,978 patent application.
In a simpler case, light generated by a non-coherent light emitting diode device is used to illuminate the desired area. In this case, the area covered is not as accurately controlled and a larger CCD array is required. Recently, however, the cost of CCD arrays has dropped substantially with the result that this configuration is now the most cost-effective system for monitoring the passenger compartment as long as the distance from the transmitter to the objects is not needed. If this distance is required, then either the laser system, a stereographic system, a focusing system, or a combined ultrasonic and optic system is required. A mechanical focusing system, such as used on some camera systems can determine the initial position of an occupant but is too slow to monitor his/her position during a crash. A distance measuring system based of focusing is described in U.S. Pat. No. 5,193,124 (Subbarao) which can either be used with a mechanical focusing system or with two cameras, the latter of which would be fast enough. Although the Subbarao patent provides a good discussion of the camera focusing art and is therefore included herein by reference, it is a more complicated system than is needed for the practicing the instant invention. In fact, a neural network can also be trained to perform the distance determination based on the two images taken with different camera settings or from two adjacent CCD""s and lens having different properties as the cameras disclosed in Subbarao making this technique practical for the purposes of this instant invention. Distance can also be determined by the system disclosed in U.S. Pat. No. 5,003,166 (Girod) by the spreading or defocusing of a pattern of structured light projected onto the object of interest.
In each of these cases, regardless of the distance measurement system used, a trained pattern recognition system, as defined above, is used in the instant invention to identify and classify, and in some cases to locate, the illuminated object and its constituent parts.
5. Optics and Acoustics
The laser systems described above are expensive due to the requirement that they be modulated at a high frequency if the distance from the airbag to the occupant, for example, needs to be measured. Both laser and non-laser optical systems in general are good at determining the location of objects within the two dimensional plane of the image and the modulated laser system in the scanning mode can determine the distance of each part of the image from the receiver. It is also possible to determine distance with the non-laser system by focusing as discussed above, or stereographically if two spaced apart receivers are used and, in some cases the mere location in the field of view can be used to estimate the position relative to the airbag, for example. Finally, a recently developed pulsed quantum well diode laser does provide inexpensive distance measurements as discussed below.
Acoustic systems are also quite effective at distance measurements since the relatively low speed of sound permits simple electronic circuits to be designed and minimal microprocessor capability is required. If a coordinate system is used where the z axis is from the transducer to the occupant, acoustics are good at measuring z dimensions while simple optical systems using a single CCD are good at measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three measurements to be made with low cost components.
One example of a system using these ideas is an optical system which floods the passenger seat with infrared light coupled with a lens and CCD array which receives and displays the reflected light and an analog to digital converter (ADC) which digitizes the output of the CCD and feeds it to an Artificial Neural Network (ANN) or other pattern recognition system , for analysis. This system uses an ultrasonic transmitter and receiver for measuring the distances to the objects located in the passenger seat. The receiving transducer feeds its data into an ADC and from there into the ANN. The same ANN can be used for both systems thereby providing full three-dimensional data for the ANN to analyze. This system, using low cost components, will permit accurate identification and distance measurements not possible by either system acting alone. If a phased array system is added to the acoustic part of the system as disclosed in copending patent application (ATI-102), the optical part can determine the location of the driver""s ears, for example, and the phased array can direct a narrow beam to the location and determine the distance to the occupant""s ears.
6. Applications
The applications for this technology are numerous as described in the copending patent applications listed above. They include: (i) the monitoring of the occupant for safety purposes to prevent airbag deployment induced injuries, (ii) the locating of the eyes of the occupant to permit automatic adjustment of the rear view mirror(s), (iii) the location of the seat to place the eyes at the proper position to eliminate the parallax in a heads-up display in night vision systems, (iv) the location of the ears of the occupant for optimum adjustment of the entertainment system, (v) the identification of the occupant for security reasons, (vi) the determination of obstructions in the path of a closing door or window, (vii) the determination of the position of the occupant""s shoulder so that the seat belt anchorage point can be adjusted for the best protection of the occupant, (viii) the determination of the position of the rear of the occupants head so that the headrest can be adjusted to minimize whiplash injuries in rear impacts, (ix) anticipatory crash sensing, (x) blind spot detection, (xi) smart headlight dimmers, and many others. In fact, over forty products alone have been identified based on the ability to identify and monitor objects and parts thereof in the passenger compartment of an automobile or truck.
This invention is a system to identify, locate and monitor occupants, including their parts, and other objects in the passenger compartment and objects outside of a motor vehicle, such as an automobile or truck, by illuminating the contents of the vehicle and objects outside of the vehicle with electromagnetic, and specifically infrared, radiation and using one or more lenses to focus images of the contents onto one or more arrays of charge coupled devices (CCD arrays). Outputs from the CCD arrays, are analyzed by appropriate computational means employing trained pattern recognition technologies, to classify, identify or locate the contents or external objects. In general, the information obtained by the identification and monitoring system is used to affect the operation of some other system in the vehicle.
When the vehicle interior monitoring system of this invention is installed in the passenger compartment of an automotive vehicle equipped with a passenger protective device, such as an inflatable airbag, and the vehicle is subjected to a crash of sufficient severity that the crash sensor has determined that the protective device is to be deployed, the system in accordance with the invention determines the position of the vehicle occupant relative to the airbag and disables deployment of the airbag if the occupant is positioned so that he/she is likely to be injured by the deployment of the airbag.
In some implementations of the invention, one or more ultrasonic transmitters and receivers are added to the system to provide a measurement of the distance from the transmitter/receiver to the objects of interest. In some of these implementations, a phased array system is used to permit the ultrasonic waves from the ultrasonic transmitters to be narrowly focused onto a particular location of an object. In other implementations, the source of infrared light is a modulated laser that permits an accurate measurement of the distance to the point of reflection. In still other cases, a focusing system is used to determine the distance to the object. Finally, in yet other cases a GaAs pulsed quantum well laser system is used to measure distance directly to a point of interest.
Principle objects and advantages of the optical sensing system in accordance with the invention are:
1. To recognize the presence of a human on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag, heating and air conditioning, or entertainment systems, among others.
2. To recognize the presence of a human on a particular seat of a motor vehicle and then to determine his/her position and to use this position information to affect the operation of another vehicle system.
3. To determine the position, velocity or size of an occupant in a motor vehicle and to utilize this information to control the rate of gas generation, or the amount of gas generated by an airbag inflator system.
4. To determine the presence or position of rear seated occupants in the vehicle and to use this information to affect the operation of a rear seat protection airbag for frontal impacts.
5. To recognize the presence of a rear facing child seat on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag system.
6. To determine the approximate location of the eyes of a driver and to use that information to control the position of the rear view mirrors of the vehicle.
7. To monitor the position of the head of the vehicle driver and determine whether the driver is falling asleep or otherwise impaired and likely to lose control of the vehicle and to use that information to affect another vehicle system.
8. To provide an occupant position sensor which reliably permits, and in a timely manner, a determination to be made that the occupant is out of position, or will become out of position, and likely to be injured by a deploying airbag and to then output a signal to suppress the deployment of the airbag.
9. To provide an anticipatory sensor that permits accurate identification of the about-to-impact object in the presence of snow and/or fog whereby the sensor is located within the vehicle.
10. To provide a smart headlight dimmer system which senses the headlights from an oncoming vehicle or the tail lights of a vehicle in front of the subject vehicle and identifies these lights differentiating them from reflections from signs or the road surface and then sends a signal to dim the headlights.
11. To provide a blind spot detector which detects and categorizes an object in the driver""s blind spot and warns the driver in the event the driver begins to change lanes, for example, or continuously informs the driver of the state of occupancy of the blind spot.
Accordingly, one embodiment of a system for controlling a vehicular system based on occupancy of an interior passenger compartment of a vehicle comprises transmitter means for transmitting waves into a space in the passenger compartment in which an occupying item might be situated, receiver means for receiving waves transmitted by the transmitter means and modified by passing through the space in the passenger compartment in which the occupying item might be situated; and output means coupled to the receiver means for affecting another system in the vehicle based on the waves received by the receiver means. Processor means may be coupled to the receiver means for generating a signal characteristic of the waves received by the receiver means and identification means coupled to the processor means for identifying an occupying item which might be situated in the space in the passenger compartment based on the signal. The identification means may comprise pattern recognition means for processing the signal to provide an identification of the occupying item based thereon, e.g., by applying a pattern recognition algorithm generated from data of possible occupying items of the vehicle and patterns of received waves from the possible occupying items.
The another system may include an inflatable airbag, in which case, the pattern recognition means process the signal into a positional categorization of the signal characteristic of the position of the occupying item based on data corresponding to patterns of received waves associated with occupying items of the vehicle in different positions. The output means may comprise means for modifying at least one of the time at which inflation of said airbag is initiated, the flow of inflator gas into said airbag and the flow of inflator gas out of said airbag in response to the position of the occupying item. Also, the pattern recognition means may process the signal into a categorization of the signal characteristic of the presence of the occupying item based on data corresponding to patterns of received waves associated with the presence and absence of the occupying item, in which case, the output means preferably comprise means for suppressing deployment of said airbag in response to either the absence of an occupying item, the presence of an out-of-position occupant or the presence of a rear-facing child seat.
The receiver means can comprise at least one receiver array, or more generally, means for converting light into electrical signals. For example, the receiver means may comprise a CMOS dynamic pixel camera, an active pixel camera and/or a HDRC camera.
These and other objects and advantages will become apparent from the following description of the preferred embodiments of the vehicle identification and monitoring system of this invention.